Fix/75 no mkdir when s3 #76
Conversation
…unnecessary and not permitted.
…unnecessary and not permitted.
lchen-2101
requested review from
hkeeler,
guffee23,
jcadam14 and
nargis-sultani
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
|
|
||
| log = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| async def upload_to_storage(lei: str, submission_id: str, content: bytes, extension: str = "csv"): | ||
| try: | ||
| fs: AbstractFileSystem = filesystem(settings.upload_fs_protocol.value) | ||
| fs.mkdirs(f"{settings.upload_fs_root}/{lei}", exist_ok=True) | ||
| if settings.upload_fs_protocol == FsProtocol.FILE: |
There was a problem hiding this comment.
It looks like the LocalFileSystem impl takes a auto_mkdir boolean which would create the directory on open if it doesn't exist, that could be passed in kwargs to open I think. If so, this might be a good way to avoid this impl specific if (which loses the abstractness which is really nice to have!). The kwarg would get ignored by S3. I'm not positive, just reading the API, but might be worth a look?
If that doesn't work, this looks good. Just was hoping to keep it all abstract but alas!
There was a problem hiding this comment.
really good suggestion, alas it doesn't work... I get an unexpected keyword argument, so unfortunately the s3fs doesn't ignore it...
closes #75